Tuesday's Teaching Tips—Evaluation and Feedback: A Spaced Education Strategy for Faculty Development

Introduction The AGGME requires faculty to participate annually in faculty development sessions. Barriers to this requirement include faculty having a lack of time and not perceiving benefits to participating. Effective evaluation and feedback are integral to resident training. Faculty often feel ill prepared to deliver feedback, and residents find accepting and recognizing feedback challenging. We provided faculty with a spaced education program via email that used cognitive theory of multimedia learning solutions in instructional design. Methods The 14-week program consisted of one microlecture and 13 skills-based teaching tips. One tip reinforcing knowledge and skills from the microlecture was emailed each week for faculty to practice in the clinical environment with trainees. Participants completed a short quiz, course evaluation, and self-reflection. The new world Kirkpatrick model was used for program evaluation. Results Fifty-two physician participants received credit for participating; 34 completed the entire course. Of the 34, 32 (94%) identified at least one effective feedback technique, and 27 (79%) were able to define evaluation and recognize observation as the cornerstone of evaluation. Out of the 15 effective feedback characteristics taught, 13 (87%) were identified. Fifty-one participants (98%) rated the program as good/excellent, 52 (100%) wanted more Tuesday's Teaching Tips programs, and the majority recognized change in knowledge and/or skills. Discussion Participants rated the spaced education program as good/excellent and were able to meet the course objectives. This teaching strategy for faculty development was well received, as it was easily accessible and implemented in the clinical learning environment with trainees.


Introduction
The ACGME requires faculty development programs to improve the knowledge, skills, and behaviors of faculty as educators. 1 However, barriers to faculty participating in faculty development are well documented and include a lack of time due to clinical responsibilities, a lack of protected time, competing continuing education needs within their own specialty, a perceived lack of recognition and financial reward for teaching, inconvenient faculty development sessions, a tendency for faculty to underestimate the need for and potential benefits of training, underestimation of the usefulness of teaching skills compared to clinical skills, and concern that teacher training is unrelated to teaching excellence. [2][3][4][5][6] Paramount to resident training is effective evaluation (assessment) of the resident's knowledge and performance as well as timely feedback to reinforce and/or correct medical knowledge and/or behavior. Effective evaluation and feedback enable resident growth in terms of formative assessment and assist with achievement of clinical competence (summative assessment). [7][8][9] However, both faculty and residents report widespread dissatisfaction regarding faculty skills in effective evaluation and feedback and resident recognition and acceptance of each when presented. [10][11][12][13][14][15] Additionally, faculty continue to report discomfort in giving feedback and feel ill prepared to accomplish this task. [16][17][18][19][20][21][22] In an effort to address faculty development regarding evaluation and feedback and to take into account faculty time constraints, we developed an innovative faculty development program entitled Tuesday's Teaching Tips-Evaluation and Feedback (TTT) that built upon programs by Matzie and colleagues 23 and Pernar and colleagues. 24 The Matzie study provided spaced education to surgical residents as part of a residents-as-teachers program to improve feedback to medical students, and the Pernar study sought to improve surgical faculty's teaching skills with medical students. In both studies, the target population received weekly emails in two cycles over 9 months with a brief statement and, in some instances, an example for implementation. Spaced education is a type of distributive practice by which information or procedures are presented and repeated in subsequent reviews to enhance long-term retention of knowledge and transfer of skills. 25 A MedEdPORTAL search using the term evaluation and feedback yielded over 1,000 publications, most of which described workshops or online modules, but none that appeared to involve an email approach to faculty development. Similarly, a MedEdPORTAL search for the term spaced education did not reveal relevant publications utilizing a spaced education approach to faculty development.
A key difference from the Matzie and Pernar programs is that ours is based on multimedia learning, in which learning occurs from words and pictures versus words only. We sought not only to provide spaced education in the form of weekly email tips regarding evaluation and feedback (words) but also to incorporate a microlecture (video, pictures, and words) establishing foundational knowledge upon which to base the spaced education emails and, further, to utilize cognitive theory of multimedia learning (CTML) solutions to address cognitive load concerns. 26 Mayer and others have suggested the importance of taking into account cognitive load to address the finite quantity of cognitive processing the mind can accomplish in both the verbal and visual channels in working memory. [27][28][29] Yet there is a balance in being able to hold the learner's attention and draw connections back to previous knowledge and experiences or information just learned (e.g., spaced education, learning and working memory tenets). The use of visual cues is a CTML solution that utilizes coloring, arrows, borders, highlighting, bolding, and other visual cues to capture or guide the learner's attention and has been shown to be effective in learning. 26 For example, whereas the Matzie and Pernar studies' emails consisted of text (words) only (top left panel of the Figure), our design (top right and bottom panels of the Figure) shows the CTML solution technique of using visual cues or signaling. This technique is Figure. Deconstructing a Tuesday's Teaching Tip: The top left panel features only a statement (words alone) versus the top right and bottom panels (using CTML solutions) as they appear in the microlecture and tip emails, respectively. The microlecture has limited words in order to focus the learner on the narration and utilizes visual cues (signaling) such as color, bolding, and borders to carry into the tip emails with their additional CTML solutions: arrows, underlining, and aligning. Abbreviation: CTML, cognitive theory of multimedia learning. Top right panel image: author created and owned; bottom panel image: public domain, created by benzoix, https://www.freepik.com/free-photo/portrait-asiandoctor-woman-cross-arms-standing-medical-uniform-stethoscope-smiling-cam_ 21129522.htm#query=asian%20nurse&position=21&from_view=keyword present in the microlecture as well as in the emails to assist with memory retrieval, along with simple text (words) and examples to reinforce the content and practical implementation in the clinical learning environment. Utilization of visual cues or signaling is one solution reinforced by CTML to reduce cognitive load in learning. Other solutions include off-loading, segmenting, pretraining, weeding, aligning, eliminating, synchronizing, and individualizing. Table 1 highlights the additional solutions we utilized to reduce cognitive load as part of multimedia learning in the microlecture and emailed tips in order to focus the learner on important content. Furthermore, our emailed tips stressed the importance of practice with trainees, and the program was shorter in duration than in the Matzie and Pernar studies (14 weeks vs. 9 months) to focus the knowledge and practice the skills.

Curricular Context and Development
We implemented TTT as an optional 14-week faculty development program on evaluation and feedback for teaching physicians across all specialties at one academic institution between February and May 2020. We utilized the framework of spaced education to first provide a foundational microlecture on evaluation and feedback and then reinforce the content through weekly emails containing skills-based tips. The microlecture and emails utilized CTML instructional design solutions to combat cognitive load, which could interfere with multimedia learning. One clinical/medical educator expert and one EdD revised core evaluation and feedback statements and examples from the Matzie and Pernar studies. 23,24 The EdD also designed the microlecture and emails utilizing CTML solutions. The microlecture (Appendix A) was recorded and made available on our institutional website for easy access by participants. Through our email system (Outlook), we emailed small pieces of reinforcing information as weekly statements (Appendix B) that were phrased as skills-based tips for faculty to implement in the clinical learning environment with trainees (students, residents, and/or fellows). Program administrators needed access to a computer, and participants used a computer, tablet, or smartphone to watch the microlecture video and read the weekly emails. We applied for CME credit through our institution, with the 15-minute microlecture as 0.25 hours and each weekly tip and practice as 0.5 hours, for a total of 7.25 CME credits for complete participation. We emailed all the teaching faculty at our institution with a program announcement and directions to enroll (Appendix C). Finally, we utilized Google Forms for faculty to register for the program electronically (Appendix D).

Implementation
The program directions (Appendix E) described the step-bystep implementation process. To those who registered, we sent a preparatory email with overall course instructions and requirements (Appendix F). We utilized our office email account, which was accessible by several members of our staff, and created a distribution list with participants' email addresses. Every Monday afternoon, we set up the tips email for the next day to be sent automatically at 7 a.m. with a read receipt request. We tracked attendance in an Excel spreadsheet using the email read receipts and added ourselves to the email distribution list to ensure that the emails were delivered each Tuesday. We completed several dry runs prior to the start of the course, ensuring that the email system and read receipts worked. Faculty were directed to email our office to document attendance if they encountered issues with their read receipt. The program was run by one EdD and one administrative staff member, requiring no more than 2.5 hours per week for preproduction to recruit, develop the enrollment list and distribution list, and prepare the tip email in advance. During the program, we spent 2 hours per week sending the emails, ensuring delivery, and tracking attendance through read receipts. The administrator spent 3 hours per week postprogram to download evaluations, deidentify evaluations, determine CMEs, and administer certificates. The EdD required 1 hour per week for postproduction, which lasted approximately 3 weeks, to troubleshoot and answer questions.

CTML Solutions Venue Explanation of Use
Off-loading: Relieve the visual channel by enhancing verbal channel.

Microlecture
We utilized limited words (reading) in the video microlecture (visual channel) and focused on narration, reducing the cognitive load on the visual channel. Segmenting (time between lessons): Knowledge is transferred better in learner-controlled segments versus continuously.
Spaced education emails Although we sent out an email each Tuesday and required the participants to read it, the email could be read at anytime before the next Tuesday and implemented at the convenience of the participants and their clinical learning environment. Pretraining: Provide essential content in advance.
Microlecture All participants were required to complete the microlecture for foundational information. Signaling: Institute the use of cues.
Microlecture and spaced education emails We used visual cues such as colors, highlighting, borders, arrows, and bolding to guide learners to key words or key words in speech or to organize images (mapping). The visual cues were consistent within the microlecture and spaced emails. Additionally, each email was organized the same: goal, example, reinforcing concept. Aligning words and pictures: Words are placed near graphics to reduce visual scanning (to focus on essential processing).

Microlecture and spaced education emails
See the Figure for an example of the words being placed near graphics for both the microlecture and spaced education emails.
Abbreviation: CTML, cognitive theory of multimedia learning. r Week 14: We sent the program evaluation link to participants. The evaluation results were deidentified by our administrative assistant and given to the TTT program directors. We sent a certificate of completion (Appendix G) to those who had participated, attested to practicing the tips with trainees in at least 11 out of the 13 weeks of TTT, and completed the course evaluation and selfreflection. We notified participants who had completed the microlecture but were unable to participate in at least 11 weeks of the course that they were able to claim CME credit for the weeks they had participated and reported their participation to our CME office.

Evaluation
Our program evaluation plan consisted of utilizing the new world Kirkpatrick model (NWKM) levels 1-3 to analyze data collected from partcipants. 30 Data collection included (1) a short postmicrolecture quiz (Appendix H) at the beginning of the program, which fit into level 2 of the NWKM, and (2) a postprogram evaluation (Appendix I) consisting of required CME questions, a self-reflection question, and an attestation to having completed the practice sessions, which fit into levels 1-3 of the NWKM. Additionally, since this program was implemented just prior to the national shutdown due to COVID-19 in March 2020, we asked if the pandemic had affected respondents' ability to participate. To assess the program objectives, we utilized the microlecture posttest quiz to determine if participants were able to define evaluation and recognize observation as the cornerstone of evaluation (NWKM level 2, learning: knowledge). The postmicrolecture quiz was given electronically through the institutional CME office. The postprogram evaluation was developed in Google Forms by program administrators and administered electronically to participants. We used descriptive statistics to analyze the data into counts and percentages and placed reflective statements into broad categories of similar characteristics for analysis.

Results
Fifty-two physicians across 13 specialties received partial or full CME credit, with 34 (65%) completing the entire program and receiving a course certificate. The 52 physician participants included two from anesthesiology, seven from emergency medicine, one from family practice, two from internal medicine, three from neurology, two from OB/GYN, two from ophthalmology, four from orthopedics, 13 from pediatrics, four from psychiatry, three from radiology, seven from surgery, and two from urology. Forty-eight percent were female. Of the 34 who completed the program, there were 11 different specialties represented, including one participant from anesthesiology, four from emergency medicine, one from family practice, two from internal medicine, two from neurology, one from OB/GYN, four from orthopedics, 11 from pediatrics, one from radiology, five from surgery, and two from urology, with 53% being female.
On the microlecture posttest quiz assessing whether participants were able to define evaluation and recognize observation as the cornerstone of evaluation, 41 of the 52 participants (79%) answered all questions correctly. On the course evaluation, 51 (98%) rated the program as good (n = 12, 23%) or excellent (n = 39, 75%; NWKM level 1, reaction: customer satisfaction), 51 (98%) reported information gained would enhance patient care or medical education (NWKM level 1, reaction: relevance), 51 (98%) had high (n = 40, 77%) or moderate (n = 11, 21%) confidence in their ability to implement changes in their teaching (NWKM level 2, learning: confidence), and 52 (100%) stated they would like to see more TTT on other topics. On the question whether the COVID pandemic had affected respondents' ability to participate, 21 (40%) answered either yes (n = 7, 13%) or somewhat (n = 14, 27%). Of those who answered a follow-up question on the ways the COVID pandemic had affected participation, the majority noted reduced contact with trainees due to canceled educational experiences or virtual interactions with trainees. One respondent had gone on military deployment, and another expressed experiencing burnout. When asked for recommendations for improvement, one participant recommended adding videos or simulated demonstrations, and two participants responded they would have liked a small, laminated card with salient teaching points as a quick reference.
To further assess the ability to describe methods of evaluation and list characteristics of effective feedback, participants who had completed the entire program and received a course certificate (n = 34) were asked to reflect on the course by answering the question "Please describe how this program has improved your overall knowledge and skills related to evaluation and feedback when working with your trainees," with a 500word limit. Table 2 lists (a) specific characteristics of effective feedback that were identified by 32 (94%) of the 34 participants and (b) methods of evaluation in which 13 (87%) of those 15 effective feedback characteristics were identified as being used (NWKM level 3, behavior: critical behaviors). The characteristics of effective feedback most often identified were "make feedback specific," "utilize a feedback sandwich," "make feedback timely," "use the W questions" for learner self-reflection, and "use and instead of but." When reviewing the reflective comments, we grouped statements into two broad categories that appeared to be logical. Many comments concerned (1) the format of TTT and (2) the perceived knowledge and/or skills gained. We identified subgroupings within these two categories. Subgroupings for the format category included the following: (1) The course format was useful for learning feedback, (2) the series of tips helped participants use the tips in busy clinical environments, and (3) participants found it helpful to learn one tip per week. Subgroupings for the knowledge/skills gained category included participants (1) citing specific techniques and principles they had learned and (2) seeing themselves as more effective providers of feedback after the course (Table 3). Participants' reflective comments assisted with program evaluation at NWKM levels 1 (reaction: customer satisfaction, engagement, and relevance) and 2 (learning: knowledge and skills, attitude and engagement).

Discussion
We developed and implemented an innovative spaced education faculty development program on evaluation and feedback that provided weekly emailed tips utilizing CTML solutions to reduce cognitive load and focus the learners on salient content. Unlike the Pernar study, 24 which found that faculty did not perceive the emailed tips as helpful, our program evaluation found that participants had high satisfaction with the format and were able to identify specific techniques for effective feedback with trainees. Faculty were very receptive to this teaching strategy, as it was time efficient and easily accessible, eliminated the need to go to a training, and offered simple skills-based strategies to practice in the clinical learning environment. Furthermore, 100% of participants reported wanting more TTT instructional sessions in the future as a method for faculty development. It is possible that our use of a foundational microlecture on which to build the spaced education reinforcement and/or the incorporation of CTML solutions encouraged faculty to practice the tips, increased their knowledge and skills, and led to perceived improvements. Additionally, our program was well received by physicians across multiple specialties and was completed over a shorter period of time than the program described in the Matzie and Pernar studies (14 weeks vs. 9 months). 23,24 Lessons Learned Given the reliance on technology, being proactive was important to ensuring a smooth implementation of the program. Setting up the tip emails in the email system in advance allowed them to be sent at the designated time each week (Tuesday at 7 a.m.). Practice runs were instrumental in discovering technical glitches with the process, such as with our use of Outlook and our network system. We discovered that if our network system rebooted overnight, it erased emails set up in advance; therefore, we had to ensure each Tuesday morning that the emails had indeed been sent. Moreover, we felt an introductory email describing how the program was designed was imperative to set expectations for the program. Second, we learned to be creative in terms of calculating the time participants would spend on this nontraditional educational activity to earn CME credit. We took into consideration the 15-minute microlecture and determined that to read the email (maybe more than once), practice the tip on one's own, and then utilize the tip with various learners and at different times during the week, 30 minutes each week was reasonable. We also allowed for partial credit if the microlecture

Comment Category Example Comments
Course Format: Course format was useful for learning feedback.
"Great format." "This course was beneficial, as it provided a well thought out overview of some teaching techniques. Having the goal and then providing the example along [with] the reinforcing concept made it easy for me to contextualize the information." "The course material provided a wonderful structural format to give necessary feedback in a constructive manner while being in an appropriate environment." "This was a wonderful program, and I recognized from the tips and lectures that my feedback to the residents was probably antiquated." Course Format: Series of tips helped participants use the tips in busy clinical environments.
"Tips were very practical and also easy to implement even in a busy chaotic workplace." "The design of this course also allowed me to easily incorporate it into my workflow and my various competing demands." "This program has allowed me to realize that productive interaction between teacher and learner can be achieved in small increments of time which can be completed in clinical space without interruption of workflow." "In particular, the regular, short tips with specific guides to practice the techniques provided me with the chance to participate in the activity despite challenges in scheduling that often preclude such opportunities." Course Format: Participants found it helpful to learn one tip per week.
"Having each of the individual tips rolled out weekly allowed me to focus on each activity and see how I can seamlessly incorporate it into my own style of teaching. As the weeks progressed and new tips became available, I found it helpful to incorporate the new activities/behaviors and layer them on top of the old ones." "These concise lessons allowed [me] to focus [on] one area of improvement every week. The course clearly demonstrated how each of the reinforcing concepts are dependent on each other." "This was a great format for this type of learning. I enjoyed the specific advice weekly and plan to work on implementing this into my practice as a junior faculty member." "I would like to see more of these type of weekly tips, as they are helpful without being a chore like some other CME formats can be." Perceived Knowledge/Skills Gained: Participants cited specific techniques and principles they learned.
"I was able to be more specific, making a point during my feedback [of] avoiding judgement.... I also used the W questions." "Reflecting... on feedback, I felt I have changed the way I have discussions with residents.... I have switched to the ask, tell, ask model of feedback.... I have also made a conscious effort to give feedback to the residents I work with as soon as possible." "The program significantly improved my skills in teaching medical students and residents.... I am now able to give brief and timely feedback, be more observant and give specific feedback based on direct observation." "I will continue trying to work on more timely and specific feedback; using 'and' versus 'but' certainly made a difference, it didn't feel like I'm telling somebody they did something wrong or they asked a wrong question." Perceived Knowledge/Skills Gained: Participants saw themselves as more effective providers of feedback after the course.
"Tuesday's Teaching Tips have improved my overall knowledge and skills related to evaluation and feedback when working with my trainees." "The knowledge gained through Tuesday's Teaching Tips has helped me become a more effective educator by enhancing my feedback skills." "After completing this course, I am confident in providing appropriate feedback to the residents and fellows in a structured manner." "The program significantly improved my skills in teaching medical students and residents, even my medical assistants." had been completed. Even though faculty signed up for the program, if they did not watch the microlecture as foundational information, they did not receive the TTT emails (the spaced education).

Limitations
Our evaluation was based on participants' self-report and not on observations of their teaching or resident evaluation of their teaching before and after implementation of the program (level 4 of NWKM) for educational outcomes. In addition, our evaluation approach was somewhat limited by the use of our institutional CME system. The quiz after the microlecture did not accurately capture correct and incorrect questions individually but rather a pass/fail; therefore, we included only 100% passing scores to assess objectives 1 and 2. There could have been more correct answers for each question if we had individual question data. For the postprogram evaluation, we utilized Google Forms to include the required CME questions and were able to add the self-reflection question and attestation. We were mindful of keeping the evaluation feasible for busy clinicians. Satisfaction questions were based on all 52 participants who received CME credit as all 52 completed the course evaluation. However, only those who both completed the course and earned the course certificate received the self-reflection questions. We did not assess participants' prior experience with or training in evaluation and feedback, which may have impacted the interpretation of the program evaluation results. Finally, we did not do a true qualitative thematic analysis of the reflective comments but rather grouped them into broad categories.
TTT is an approach that addresses faculty development barriers of faculty time constraints, convenience in attending, and perceived benefits. The program was easy to implement, required minimal staff time, and used the Outlook email system for program delivery as well as an Excel spreadsheet to track participation. The program has broad applicability across multiple medical specialties in an academic medical center with no obvious challenges to generalizability. The tips can be integrated into an existing faculty development curriculum or utilized as a stand-alone program. Next steps involve a qualitative analysis of the participants' reflective statements. Future research or evaluation should include the behavioral impact of TTT on faculty skills as well as the impact of TTT on learners.